Papers with Elastic Weight Consolidation
Unsupervised Pretraining for Neural Machine Translation Using Elastic Weight Consolidation (P19-2)
Copied to clipboard
| Challenge: | Neural machine translation (NMT) uses sequence to sequence architectures, but requires a huge amount of parallel data. |
| Approach: | They use Elastic Weight Consolidation to regularize weights of two language models . they then fine-tune the model on parallel data to avoid forgetting the original task . |
| Outcome: | The proposed method achieves BLEU scores similar to the previous work, but is slower and requires less training data. |
Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation (N19-1)
Copied to clipboard
| Challenge: | Neural Machine Translation (NMT) performs poorly without large training corpora. |
| Approach: | They propose a machine learning method that retains the majority of general-domain performance lost in continued training without degrading in-domain. |
| Outcome: | The proposed method retains the majority of general-domain performance lost in continued training without degrading in-domain performances. |
Continual Learning for Natural Language Generation in Task-oriented Dialog Systems (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing neural approaches for natural language generation are typically developed offline for specific domains. |
| Approach: | They propose a method to expand NLG knowledge incrementally to new domains . major challenge is catastrophic forgetting, meaning a model forgets the knowledge it has learned before . |
| Outcome: | The proposed method outperforms other methods by effectively mitigating catastrophic forgetting issue. |
Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive models trained with maximum likelihood estimation suffer from exposure bias, i.e. the discrepancy between ground-truth prefixes used during training and model-generated prefix at inference time. |
| Approach: | They propose to use Elastic Weight Consolidation to better balance mitigating exposure bias with retaining performance. |
| Outcome: | The proposed method significantly outperforms maximum likelihood estimation and scheduled sampling baselines on four translation datasets. |
Improving the Quality Trade-Off for Neural Machine Translation Multi-Domain Adaptation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Building neural machine translation systems to perform well on a specific target domain remains a challenge. |
| Approach: | They propose to train a single NMT system per language pair that performs well across multiple domains. |
| Outcome: | The proposed approach improves the Pareto frontier on this task. |